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I. INTRODUCTION 



"his report summarizes all Year 11 National Assessment of Educational 
Progress activities performed under contract Nos. 02-79-16141 and 
02-80-18218 by Wstinghouse DataScore Systems. The general time 
frame for DataScore's contract activities runs from March 1979 
through October 1980 (with the exception of the Year 10 Art Scoring 
activities which are projected for February 1981 completion). 
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PREPARATION AND PRINTING 



Figure 1 summarizes preparation and printing specifications for Ages 
9, 13, 17, and 17 Outrof-School documents. (Note that Age 17 
packages and supplements were printed in sufficient overage to Supply 
the Out -of -School Adminis tra tion . ) 

In addition to those documents listed in Figure 1, nine (9) inserts 
were typeset and/or printed for inclusion with certain packages 
(indicated by "I" after package number on the figure). A summary of 
these inserts can be found in Figure 2. 



General procedures followed for the preparation md printing process 
are outlined below: 

1. NAEP printing coordinator sent two (2) copies of the 
finalized and approved document mock-up to the DataScore 
coordinator. 

2. After DataScore coordinator's approval of package proof, 
proof sent to NAEP and copy to RTI. 

3. Corrections from NAEP were phoned to DataScore coordinator, 
followed by memo. 

4. Final corrected page proofs were provided to NAEP. 

5. DataScore internal document control approval was obtained. 

6. Written authorization to print was received from NAEP. 

7* Document was scheduled for printing at DataScore. 

8. Upon age class completion, one "camera-ready 11 package proof 
of each package was produced for NAEP files. 

A suggestion for improvement of preparation procedures from the NAEP 
printing coordinator involves irregular production and shipment of 
proofs from DataScore. Every effort should be made to: 1) produce 
proofs with an "even" effort from the darkroom to avoid "mass 
mailings"; 2) mail proofs to NAEP as available instead of holding for 
additional proofs; and 3) phone NAEP printing coordinator when 
shipments are made. 

Age 13 preparation hours were underprojected due to delivery of 
non-finalized mock-ups to DataScore for typesetting. This problem 
resolved irself in subsequent age classes. 

Printing of age class documents was targeted for completion one month 
prior' to age class distribution to ensure adequate packaging time 
allowances. Covers were continuous form, package covers identical 
within age classes. Computer-slugged information included package 
number and consecutive identification numbers. Ink colors were 




determined by NAEP. FEDAC clearance numbers for all documents exce 
the supplements were provided prior to document preparation. 

Age 17 supplement p eparation and printing encountered problems due 
to the tight schedule, all supplements being prepared and printed 
within a two-week time frame. Overtime was required to complete 
printing on schedule. FEDAC clearance numbers were stamped on the 
documents after shipmment to RTI. 
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FIGURE 1 
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Figure 1, continued 
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(I) - Insert Required 

P.Q. = Principal's Questionnaire 

B.Q. - Background Questionnaire 

1700S * Age 17 Out-of-School Assessment 
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FIGURE 2 
YEAR 11 PACKAGE INSERTS 

Age Package // Item 9 Description # Pages 

7 Dictionary page (plot- plunge) 

7 Table of Contents 
5 Bridgso 

8 Index (Conquistadores-Education) 
5 Dictionary page (H-haet) 

10 Index (oil-yurts) 

8 Table of Contents (The Earth. . .) 

11 Dictionary page (Toledo-tonsorial) 
5 The Galapagos Islands 



All 
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III. PACKAGING AND DISTRIBUTION 



Package inserts were inserted into packages in appropriate locations 
per NAEP instructions. Shipping memos were prepared for each 
shipment per RTI specifications, documents were packaged, and each 
shipment was quality controlled for accuracy. Documents were 
packaged in standard hardshells with appropriate quantities of empty 
hardshells forwarded for document return. Distribution schedules 
were provided by RTI. Documents for each age class were bulk-shipped 
to District Supervisors per address and shipping specifications 
received from RTI. 



! 
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IV. RECEIPT CONTROL 



The year 11 receiving process was conducted by handscoring personnel 
during the following time frames: 



ACE FROM TO 

13 10/22/79 1/28/80 

9 01/18/80 3/24/80 

17 03/17/80 6/06/80 

The opening and checking procedures are demonstrated in the receiving 
specifications included as Appendix A of this report. 

The following table shows the n-counts for each age group: 
— - TOTAL RECEIVED NONELICIBLES TOTAL PROCESSED 



13 41,684 no 

9 29,185 82 

17 36,186 55 



41,574 
29,103 
36,131 



The average opening rate was 3.2 hours per hardshell. For Ages 13 
and 9, all receiving was completed before scoring started. In Age 
17, however, receiving was only 60% completed when scoring began 
because of material still arriving. 

The final step of Receiving is lagging. Before being scanned, a 
batch consists of 36 stacks with an approximate n-count of 4,500. 
Only one package number is included in a batch except for the 
combining of late arrivals. Each stack has a batch/stack header 
identifying the program, hatch, stack and special service numbers. I 
logging record is initiated for location reference and includes the 
batch number, stack number, PSU/school number and number of packages 
)n a PSU. r 6 

Problems encountered during the Year 11 receiving process include: 

1. ID/PSU/School nunbers incorrect or not gridded on 
supplements . 

2. Incorrect supplements given during administration. 

3. Regular/follow ups not gridded or gridded incorrectly on 
packages. 

4. Regulars and follow ups not separated on worksheets. 

5. Background questions on packages not completed. 
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6. P.Q.'s for schools with "800" numbers (split sessions) 
recopied on separate P,Q. 

n . G.A.S.'s with incorrect information such as PSU/School and 
I.D. numbers. 



Suggestions for Year 13 include: 

1. Eliminate the hardshell cover sheet since the information 
it contains is present on the school worksheet, o 

2. Devise a method of letting DafaScore know when a shipment 
is complete. 
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Year 11 handscoring was fairly smooth with delays in starting due to) 
completion of receiving. Staffing was more stable than in Year 10 / 
with approximately nine scorers, all having Bachelor's Degrees. •Tfie 
subject area of reading and literature includes several types di 
scoring: character inference, mood inference, emotional, evaluating, 
analytical, qualities, and' general response. Each type of scoring 
has a separate guile. The staff was not separated into scoring 
groups because packages were not arranged by like scoring types. 

There were a total of 21 items of which 14 were age crossovers and 
seven were unique to an age class. A list of items and scoring 
methods can be found in Figure 3. 

The following are dates in which the various stages of scoring 
occurred • 





Age 


13 


Age 


9 


Age 


17 




From 


To 


From 


To 


From 


To 


Guide Refinement 


1/4 


1/24 


3/18 


3/19 


4/14 


4/18 


Training 


1/16 


1/25 


3/18 


3/19 


4/14 


4/18 


Scoring 


1/28 


3/26 


3/31 


4/1 


5/16 


7/11 



Guide refinement and training was conducted by NAEP analysts. The 
preliminary guides were developed by the analysts and the reading and 
literature consultants. Items and guides were presented to 
DataScore's senior handscoring staff. Problems were discussed and 
guides finalized. A training packet containing item' statement , 
scoring guide with examples, and a set of unscored training papers 
were given to the scorers. . fc 

Scoring was done by package followed by another package that had the 
same scoring types. Most items were scored once with a 10% sample 
second scored as a quality control. Because one table leader was 
involved with processing full time, it was difficult for the other 
table leader to keep pace with quality control and get results back 
to the scorers. Also, scoring was interrupted to a small degree by 
rescore preparation. 

Year 02 was rescored for a replication study. NAEP provided a tape 
and list of ID's to be scored. Stacks were retrieved from the 
mezzanine, ID f s located and checked against the list. Scoring was 
done on punched cards produced from the tape. The rescore papers 
were intermixed with Year 11 papers- lor scoring. Open-ended clerical 
items received a 10 percent rescore and a 100 percent rescore was 
done on open-ended literature items. 

One large concern is the printing of packages months before scoring 
occurs. Some puide changes require the use of imaginary gridding 
ovals. A scorer must estimate where the oval would exist and grid 
accordingly. If the grid is not on center, the scanner does not 



-10- 

13 



pick up the data and editing must enter the information which can be 
very time consuming. If preliminary scoring categories and/or guides 
are not drafted at the time the materials are printed, it might be 
advisable to include some additional ovals." Although excessive ovals 
may slow the processing in scoring and editing, the full impact is 
not as great as having to work with imaginary ovals. 
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NAEP NUMBER 



H-A05000-B1B-1 , 


2, 


H-A65000-B1B-1, 


2 


A-202002-22B-1 




H-8A2000-B1B-1, 


2, 


H-8A1000-B1B-1 , 


2, 


H-AA3000-B1B-2, 


3 


A-20202A-22B-2, 


3 


H-A02000-B1B-2, 


3 


H-A67000-B1B-2, 


3 


H-A01000-B1B-2, 


3 


A-200005-22B-2 




H-A6AO00-B1B-2, 


3 


H-AA1000-B1B-2, 


3 


H-A06000-B1B-2, 


3 


H-A62000-B1B-2, 


3 


H-A13000-B1B-2, 


3 


H-A69000-B1B-3 




H-A70000-B1B-3 




H-A20000-B1B-3 




H-A21000-B1B-3 




A-200008-22B-3 





INF = 
CHAR = 
MOOD = 
EMOT = 
EVAL = 
ANAL = 

j QUAL = 

G.R. ■ 

O 
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FIGURE 3 
YEAR 11 HANDSCORED ITEMS 
TITLE 

3 Nuts! 

Old Dog 
As The Cat 
3 Good Poem 
3 Good Story 

Travels with Charley 
Rodeo 

Somebody 1 s Son 
Mother To Son 
Somebody 1 s Son 
Check 
The Door 

The Old Tomato Man 
Good Dog 
Letter To Nancy 
Cow-Tail Swi-tch 
i was you 
Lines For Winter 
One Of These Days 
The Kite-Maker 
Into My Heart 

nf erence 
Character Inference 
Mood Inference 
Emotional 
Evaluating 
Analytic 1 
Qualitie 

General Responding 
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AGES SCORING 

9, 13, 17 EVAL 
9, 13 EMOT 



9 


INF 


9, 13, 17 


QUAL 


9, 13, 17 


QUAL 


13, 17 


MOOD 


13, 17 


MOOD 


13, 17 


G.R. 


13, 17 


EVAL 


13, 17 


CHAR 


13 


G.R. 


13, 17 


G.R. 


13, 17 


CHAR 


13, 17 


EMOT 


13, 17 


G.R. 


13, 17 


EVAL 


17 


ANAL 


17 


EVAL 


17 


EVAL 


17 


ANAL 


17 


G.R. 



VI. PROCESSING 

Year 11 processing was also fin improvement over Year 10. The 
shunting of handscored items proves to be beneficial for everyone 
except programming, as it adds one more scan and edit program to the 
project. Shunting during the first pass does allow for faster 
processing of multiple choice questions and therefore production of 
the n-count tape for RTI which is used to compute weights. RTI s^nds 
a weight Cape to DataScore for use in the reformat program. 

Two procedures were improved and should be continued. 

1. Modify condition code logs. The old method consisted of 

0 verifying each ECC record in editing. A "2" error in the 
previous edit specifications indicated a non ECC which had 
to be verified and then passed through the post-edit by 
submitting two (K&T) correction cards containing pound 
signs. This was a difficult task for editing and caused 
numerous correction card entries. 

Receiving still keeps ongoing lists of all partial 
administrations in the new system. Listings (sorted by 
package) are produced after the pre-edit and each 
post-edit. The batch, stack, serial number, and the entire 
response are printed. The receiving log is verified with 
the printout and discrepancies are resolved with correction 
cards. 

2. Inclusion of data on reformat tape from GAS (Group 
Assignment Summary). A clerk transcribes the data from the 
top portion of the GAS and key enters the data. A utility 
program is used to create a file. The reformat program 
accesses the file, verifies the key, and merges the data 
onto the K-type 13 reformat record. The one problem with 
this procedure concerns the DS not designating a regular 
(R) or follow-up (F) administration. 

Handscoring personnel continued to be involved with processing and 
coordination. DataScore received specifications from NAEP for pages 
to be shunted, tape positions, forcings and consistency checks, and 
response ranges for first and second passes. All materials tre' 
reviewed by handscoring to insure any changes in scoring categories 
that af fee t tape posi tions and response ranges . After the review, 
scoring keys and response range keys are coded and delivered to the 
team leader in Programming. The scanning and first machine pass edit 
programs are written, tested with a NAEP and DataScore test batch, 
and checked by DataScore's Quality Control Department- Once the 
programs have been approved by the Quality Control Department, NAEP 
conducts their own'Quality Control and gives authorization to process 

The first pass of machine processing involves scoring all multiple 
choice items and shunting of handscored items. Batches are arranged 
by package for easiei reference and to keep the same open-ended items 
together. Editing resolves all errors, checks the condition code 
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list against the condition code log from receiving, and the batch is 
submitted for a post-edit. If the post-edit is not clean, further 
corrections are made and condition code lists and logs are checked 
again for possible updates. Once all post-edit batches are clean, 
the tapes are saved for the merge program and an n-count tape is 
produced and sent to RTI for weight calculation. 

Second pass machine processing contains a second scanning and edit 
program for handscored items only. Again, batches are arranged by 

Dacka&P. Thp samp nrorPsS of qranni r»o a-nti oft l* f ^ 0 <*~t* 

editing is done in handscoring) as first pass is conducted omitting 
the condition code check and n-count tape. 

The reformat program merges first and second pass batches, RTI ! s 
weight tape, and the GAS file mentioned earlier, A reconciliation of 
batches is completed and additional comments may be made to the 
condition code listing before a final tape is produced and sent to 
NAEP. 

Rescore processing involves the revision of specifications of tape 
positions, response ranges, and forcings and consistency checks for 
items scored in previous years. These are delivered to the 
programming team leader along with a tape supplied by NAEP, The tape 
contains information such as ID numbers and PSU/schools to be 
rescored. From the tape, cards for each ID gre punched and delivered 
to handscoring on which to do the scoring. Scan and edit programs 
are written, tested, and checked by Quality Control, After the 
pre-edit and editing, clean post-edits are merged with the NAEP 
tape. All missing ID's are resolved and the final tape is sent to 
NAEP. 

Two concerns should be mentioned at this time: 

1. The Varidoc tape procedure seems to demand a lot of labor 
hours. Alternative procedures should be investigated 
before 1982. 

2. The edit program for the second machine pass (handscored 
items) appears to be holding up processing by a matter of a 
few weeks. Finalized tape positions, response ranges, and 
forcings and consistency checks are known only after 
scoring has started. Also, there seems to be some conflict 
with various stages of programming in overlapping ages. 
Shunting requires the extra program and we should allow 
adequate time for it to be done. 
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VII. OUT-OF-SCHOOL: GENERAL DISCUSSION 

The OOS project ran very smoothly and created no conflicts with the 
in-school schedule as it oc cured inmiedia tely after. Printing 
consisted of the Student Background Questionnaire which was shipped 
to RTI for distribution on May 9. Because NAEP did not receive the 
OMB clearance number until May 12, it was decided that DataScore 
should print and have RTI enter the number on all BQs. 

Receipt Control was conduc ted from June 9 through August 11 with a 
total of 658 student envelopes received of which 7 were determined to 
be non-el igibles . Each respondent was allowed to take up to 3 
packages and the BQ. Envelopes were opened as they arrived and 
checked to provide RTI with immediate feedback of any errors or 
missing information. Weekly updates of respondents received and 
problem tallys were sent to RTI. The only delay experience was 
waiting for RTI to receive missing information on BQ tailsheets. 
Procedures for receipt control were identical to in-schools with the 
addition of checking package numbers and order of administration with 
the BQ. 

Scoring took place July 9 through August 12 using the same staff as 
in-schools. The items and guides were identical to 17 in-schools 
eliminating guide refinement, training, or guide review. Most of the 
responses were quite short so that scoring didn't' take as long as 
expected. Only one computer pass was made but handscored items were 
shunted and stored with 17 in-school items. Pre "dpd post edits were 
QCed August 19 and 20 and the final leformat tape was mailed October 
16. 
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VIII. 



ART SCORING PROJECT 
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All materials were retrieved from the mezzanine prior to scoring and 
arranged in ID order according to Age Class, Package and Year. A 
check of IDs against list supplied by NAEP was made to determine 
missing IDs and extra packages (which were most often birthdates out 
of range) that had not been taken out of the stacks. 

Guide refinement took place from September 15 through September 25 
and included a NAEP analyst, art consultant, and handscoring 
coordinators and manager. The refinement took an additional five 
days over the forecast (totaling almost two days per guide) which 
delayed training and scoring. In retrospect, it would have been 
valuable to have the art consultant spend more time with the NAEP 
analyst prior to refinement. Also, an objective statement and 
detailed scoring rationale for each item would have been helpful 
throughout the project. 

Training occurred September 25 through October 3 and was led by the 
NAEP analyst and the art consultant. The same materials that were 
used in refinement were used for training and consisted of a scoring 
guide with examples and sample papers to score. One item took 1-1/3 
days of training on the average. 

DataScore's art consultant had a degree in art with teaching 
experience.^ She also doubled as a table leader for both teams and 
was responsible for making and relaying scoring decisions and 
conducting quality control. All scorers had a Bachelor's Degree. 
Following is a list of items and averages per team. 

Team 1 (4 scorers) 

1-101005-62A-1,2,3 Two Flowers 

1-301008-62A-1,2,3 Necklace 

l-202042-62A-2,3 Big B 

Team 1 averaged 12.3 packages per hour or about 2 minutes per item. 

Team 2 (4 scorers) 

l-302006-62A-l,2,3 Six Sketches 

1-501012-62A-1,2,3 Picasso Horses 

l-304014-62A-2,3 Angry Person 

Team 2 averaged 8.5 packages per hour or about 2.6 minutes per item. 
Each team scored approximately 40,000 items. 

Scoring started September 26 with Team 1 finishing December 19 and 
Team 2 finishing on January 15, 1981. Scorers constantly changed 
year and age packages to maintain uniform application of scoring 
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across ages and years. It took scorers a full month to reach the 
peak production rate. The subjectivity of some of the scoring 
catagories helped cause the slow start and rates didn't increase as 
expected. This was a cause of scorer frustration. A particular 
problem for Team 1 was scoring the cohesive and vivid categories on 
"Big B." This was resolved by having only two scorers score these 
categories, which increased the team's scoring rate and consistency. 
Team 2 was slower because of the amount of gridding required for "Six 
Sketches." 

Quality control was conducted ovei 7 weeks and included all age 
groups in both years. Team 1 showed an overall agreement rate of 95% 
and Team 2 had a 92.8% result. Following is the percentage of 
agreement by exercise: 



Team 1 



Exerc ise 



% Agreement 



Two Flowers 



95.2 



Necklace 



94.0 



Big B 



95.3 



Team 2 



Exerc ise 



% Agreement 



Six Sketches 



95.0 



Two Horses 



89.0 



Angry Person 



94.5 
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IX. MAINTENANCE AND STORAGE 

All Year 11 NAEP materials were added to NAEP inventory in mezzanine 
of DataScore warehouse. Materials will be added co computerized 
inventory early in 1981. 

DataScore's warehouse storage space was increased from 1,260 square 
feet to over 2,000 square feet to accommodate the crowded storage 
situation. Additional steel shelving was installed for package-stack 
storage. 
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X. PROJECT MANAGEMENT 



DataScore organization A changes included: 

April 1980: Acting Handscoring Manager took position at another 
division processing site. New DataScore Project Manager and 
Handscoring Manager announcements were presented. 

August 1980: One Handscoring Coordinator left project to resume 
graduate studies. An art consultant was retained to assume 
coordinative, as veil as consultant responsibilities for 
upcoming art scoring project. 

December 1980: Acting Project Manager assumed position in 
another department. A new Project Manager, with extensive NAEP 
background, was named as a replacement, project transition to 
occur in January 1981. 

NAEP Management Meetings were attended by DataScore personnel in 
December of 1979 (Raleigh), January (Raleigh), March (Raleigh), and 
June 1980 (Iowa City). 

An Out-of-School planning meeting was held in April 1980. 

In June of 1980, DataScore personnel travelled to Denver to discuss 
ballpark costs for subsequent assessment year projected 
specifications. 

In December of 1980, a Year 11 general debriefing and preliminary 
Year 13 planning meeting was held in Iowa City. Conferences and 
reporting tasks also included: 

Quarterly Reports (1981) 

Cost Projections 

Schedule Revisions 

Contract Administration (three modifications to Contract 
No. 02-79-1641; one modification to Contract No. 
02-80-18218.) 

Project management concerns expressed by NAEP: 

1. Heavy involvement of Handscoring Manager in non-NAEP 
> handscoring during first four months of 1980 caused 

[. v proDlems in communication and subsequently impacted the 

schedule . 

2. The DoDDS-80 project had to be monitored closely to avoid 
NAEP schedule interference. 

3. Contract modification to 1980 contract was not provided 
prior to task completion. 
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YEAR 11 

APPENDIX A Age Class 2 

NATIONAL ASSESSMENT 
Receiving Specifications 

TERMS 

PSU = Primary Sampling Unit = A PSU is a 2 digit number which identifies 

the geographical location of the sample. 

DS = District Supervisor = The country is divided into 13 parts. A 

district supervisor is assigned to each part. 

EA = Exercise Administrator = EA's are hired by a DS to administer the test. 

RTI = Research Triangle Institute = Located in Research Triangle Park, North 

Carolina (center of a triangle formed by 
the cities of Raleigh, Durham, and Chapel 
Hill). RTI is the NAEP subcontractor 
responsible for sampling and 
administration. 



RECEIVING PROCEDURES 

1) Hardshells for approximately 10 PSU's will oe received by bus each week. 
These may be complete or partial PSU shipments. 

2) When stock control delivers the hardshells they will notify the person in 
charge of NAEP Receiving. 

3} The person in charge of NAEP Receiving will enter the date and number of 
hardshells received on the Receiving Checklist. (See Attachment 1.) This 
checklist must be ke t current. 

4) DS's should have written the PSU number on the too of the hardshell. Enter 
this number on the checklist. If the PSU number is not written on the 
return card, open the hardshell to find the number and enter it on the card. 

5) The hardshells received should be stacked by PSU and set aside for opening. 
RTI INCOMING FORMS 

Group Administration Schedule Forms (yellow tear-off s)= 

Identifies grade, sex, birthdate, and race for each ID/pockage/school PSU. 
This is a yellow strip of paper or tear-off. 

Hardshell Cover Sheet = 

Inventory of the quantities of each package sent for a given hardshell. 
(See Attachment 2.) 
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Package Assignment Summary = - ° 

Lists the schools included in each PSU. A range of packages are also listed 
for each school. (See Attachment 3.) 

School Worksheet = 

Inventories the packages *nd their quantities tested in the school. 
Administration problems are written in the section at the bottom of the 
page. (See Attachment 4.) 

DATASCORE OPENING FOR KS ' 

Problem Tally Sheet = 

Describes certain errors committed by a DS in returning the hardshell. (See 
Attachement 5. ) 

School Header = 

Identification Header for DataScore processing. (See Attachment 6.) 

School Worksheet Tally = . 

Inventories the number of each package for each school. (See Attachment 
10.) 

Condition Code Log = 

Identifies and describes problems in administration. (See Attachment 11.) 

OPENING PROCEDURES 

1) Choose a PSU. Be sure to pull all the hardshells which are present for that 
PSU. Choose those PSU's in order of receipt. 

2) Pull the Package Assignment Summary for the PSU. (See Attachment 3.) 

3) Start Condition Code Logs - one condition code log will be used for each 
package number. (See Attachment 11.) 

4) Start a Problem Tally Sheet. 

5) Open the first hardshell of the PSU. Refer to the chart for the contents 
and order of an unopened PSU (see Attachment 7). Each hardshell should con- 
tain a hardshell cover sheet. Take out the hardshell cover sheet and set it 
aside. If a cover sheet was not enclosed, enter an error on the Problem 
Tally Sheet. Fill out a cover sheet for the hardshell. 

6) The next document in the hardshell should be a school worksheet. Read Sec- 
tion F to alert .you to administration problems which you will encounter 
later in the hardshell. Check to make sure there is a "Final Group Size" 
listed on the worksheet. 

A) A Principals' Questionnaire should be enclosed for each school. If the 
, appropriate PQ is not enclosed, the DS should have written a note 
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explaining when it will be transmitted. (If no note is present, mark 
and error on the Problem Tally.) 

B) Make sure the PSU/School number is entered and gridded. 

C) File the PQ(s) in PSU/School order. 

D) If the PQ(s) will be coming later, insert a sheet of gold paper with the 
PSU/School number written on it. When the PQ comes in take out the gold 

^sheet. 

7) The Group Administration Schedules (GAS's) should follow the worksheet. In 
some cases the GAS's will be tucked inside the first package of each package 
set. Set them aside. 

8) The package sets for that school should follow the GAS's. Each set of 
packages should already be turned in the hardshell. 

9) Take out the first set of packages. The PSU and school numbers are written 
on the front of each package - Section #8. The first two entries are the 
PSU. The last three entries are the school number. Be sure that you have 
the PSU and school to match the school worksheet you are working with. Make 
sure "all the packages in one set are the same package number. 

10) Count the packages in each package set and check off that distinct package 
Lumber on the School Worksheet with a red pen. If the counts do not match, 
check the GAS for that package set. If the GAS agrees with your count, 
change the School Worksheet in red and enter an error on the Tally Sheet. 
If the GAS does not agree with your count, see the Receiving Coordinator. 

11) Check the information from the GAS to the front of each package-Section 3-6. 

A) Find the GAS for that package number. 

B) Check to make sure the same PSU and school number is written on the GAS, 
school worksheet and EACH package. 

C) Find the ID number on the GAS that corresponds with the ID on the 
package. 

D) Check the data entered in Sections 3-6 on the cover of the package. 

1) Column 3 = Grade 

98 = Ungraded Class 

99 = Special Education 

2) Column 4 = Sex 

1 = Male 

2 = Female 
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3) Column 5 = Birthdate 

13 year olds = 01-12, 1966. 

Make sure the year is in range. If not, the student is not eli- 
gible. (See Section 1 2d. ) 

4) Column 6 = Race 

1 = White 

2 = Black 

3 = Spanish Heritage 

4 = American Indian or Alaskan Native 

5 = Asian or Pacific Islander 

6 = Other 

E), If you find a discrepancy between the yellow GAS tear-offs and the cover 
entries, change the entry on the package. Enter an error on the Problem 
Tally. 

12) Check to see that one oval is gridded in the Package Wide Condition Code 
(PCO). The PCO is located in the lower left corner of the cover page. The 
ovals are marked N (normal), P (partial), NR (no response), and NE (not 
eligible). . - - - 

A) When the PCO is an N, this means that the administration was normal. 

B) If the PCO is a P, there should be a note from the DS either on the 
package or worksheet explaining why the administration was partial and 
which items were not completed. If this note is only on the worksheet, 
transfer it to the front of the package and enter an error on the , ~ 
Problem Tally Sheet. 



1) Enter the ID and note the condition code log for that package. 

2) If there is no note, consult the Receiving Coordinator to resolve 
the problem. If you must change the P to N, enter an error on the 
Tally Sheet. 

C) If the PCO is an NR: 

1) All items must be no response. 

2) If any item has been responded to, check the eligibility of the stu- 
dent and change the NR to N or NE (see D below) and enter an error 
on the Tally Sheet. 

D) If the PCO is an NE: 

1) The DS should have written the problem on the front of the package. 

2) When no note is written, look for the problem. 

a. Look at the birthdate on the GAS to see if it is out of range 
(00R). 

b. Consult the Receiving Coordinator for assistance in resolving 
the problem. 
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c) If no note is written on the package or worksheet, mark an error 
on the Tally Sheet. 

d) Write a note on the front of the package (if no note is written) 
explaining why the package has been deleted. 

3) Pull the package out of the assessment. 

4) Make sure the PSU and school numbers written on the front of the 
package are correct, 

5) Correct the package count on the school worksheet, unless the DS has 
already done so. The Hardshell Cover Sheet count should INCLUDE the 
deleted package. 

6) On the school worksheet, make a note of the ID number, package 
number and the reason for deleting the package. 

13) Background questions are on the tailsheet of each package. (See Attachment 8.) 

A) A school may refuse to have the students answer these questions. 

1) The DS must write a memo or make a note on the school worksheet. 

2) If there is no note, check the list of school refusals which was 
sent by RTI. 

-3) Notify the Receiving Coordinator of any refusals which are not on 
RTI's list. 

B) If any or all of questions 1-9 (on the back page) are blank, code the 
IDK oval. If any or all of 5-9 are blank, enter an error on the 
Problem Tally sheet. 

C) Question 9 asks ifl which states the student lived on his/her 9th 
birthday. 

1) If "In the U.S.", check the following. 

a) See that the oval is gridded in the proper part. A state should 
be written in the proper blank. 

b) Grid the state code in the column at the right. (See Attachment 
8 and 9.) 

c) If a territory is written, check the list of U.S. territories in 
Attachment 9. 

1) If the territory is listed, grid "OT" in the column at the 
right. 

2) If the territory is not listed as a recognized territory, 
check with the Receiving Coordinator. 

d) If a city is written on the line: 

1) See if that city has been listed in conjunction with a state 
name on other packages in that school. If so, code that 
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state. If not, look- in an atlas to see if a city by that 
name exists in the state in which the school is located 
(found on Package Assignment Summary). 

2) Enter an error on the Tally Sheet. 

If the line is blank: 

1) Grid the oval labeled BL (3rd from the bottom). 

2) Enter an error on the Tally Sheet. 



2) If "Outside the U.S.", check the following. 

a) See that the oval is gridded in the proper part. 

b) A country should be written on the line. If so, code "0C" in 
the column on the rights 

c) If the line is blank, code BL (at the bottom of the page) and 
enter an error on the Tally Sheet. 

3) If IDK is gridded, do not cpde anything in the column on the right. 

14) When you have finished a set of packages, complete a school header. There 
will be header for each package set. Enter the following information. (See 
Attachment 6.) 

A) 2~digit PSU. 

B) 3-digit school. 

C) NCNT — total number of packages (3 digits). 

D) Number of packages ~ enter only counts for that particular package set. 

15) The packages are now ready for a preliminary handscoring phase. At this 
'time, enter and grid all embedded ID's, handscore the clerical scoring 

items, condition code any affected items and check to make sure the inserts 
have been removed from the packages. A list of inserts and their locations 
f ollov/s. 
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Age Class II 


Inserts 






Insert § 


Insert Description 


Package 


Item # 


Page # 


1 


Dictionary Page 


4 


7 


14-15 




plot/plunge (1 P.) 






16-17 


2 


Table of Contents (4 pp.) 


11 


7 


3 


Bridges (4 pp.) 


12 


5 


8-9 


4 


Index - Conqui stadores/ 


12 


8 


24-25 




Education (1 p.) 








5 


Dictionary Page 


13 


5 


14-15 




H/haet (1 p.) 









16) Place the package set in a cardboard box. Each package number is boxed 
separately. For example, there will be a box of package l's, a box of 
package 2's, etc. One box will contain several PSU's. The PSU/School must 
be kept together under the school header. Turn each school in the box. 

17) Repeat steps 1 to 16 for all package sets in the school. 

2u 
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18) At the completion of a school, be sure you have accounted for all the pack- 
ages listed on the school worksheet. 

19) Using the school worksheet, check off the school on the Package Assignment 

Summary. Also make sure that the packages assigned to that school have been 

tested. Notify the Receiving Coordinator if there are any discrepancies. 

If there were no packages to be administered in a school, write "NN " by the 
school number. J 

20) Again using the school worksheet, check off the number of packages returned 
against the quantitites listed orr^the hardshell cover sheet. Make your 
corrections in red and enter an error on the Problem Tally. 

21) If there are any problems which can't V solved without calling RTI fill 

OU L a ^ eCe !u in9 / r0blem Sheet * Turn this in t0 the Receiving Coordinator 
with the other forms when you have completed the PSU. 

22) Repeat steps 1 to 21 for each school in the PSU. 

22) If the PSU is complete, check "complete" on , the Problem Tally. If the PSU 
. is not complete, check "partial" on the Prdblem Tally. 

) 

24) Xerox 2 copies of the Problem Tally Sheet. \Xerox 1 copy of the School 
Worksheet if any changes have been made. Xerox 1 copy of the Hardshell 
Cover Sheet if any chagnes have been made. 

25) At the completion of a PSU, paperclip the following together: 

A) Package Assignment Summary ... 

B) Hardshell Cover Sheets 

C) School Worksheets 

D) Problem Tally Sheet 

E ) Deleted Packages 

F) Xeroxes 

Turn the finished forms in to the Receiving Coordinator. 

26) Separate the GAS's by package. File them by PSU/School. 

27) The keypunch form, "National Assessment School Worksheets," will be filled out 
by the Receiving Coordinator. (See Attachment 10) 
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- Yr 



Age 

RECEIVING CHECKLIST 



PSU 


Date 
# of pieces 


RTI 
notif. 


Date 
# of pieces 


RTI 
notif. 


Date 
# of pieces 


RTI 
notif. 


Completed 
Shipment 
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Receivers should initial (in red) a shipment after it is opened. 
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ATTACHMENTS 3 and 4 

OMMITTED DUE TO CONFIDENTIALITY 

\ 

\ 



Hardshell Cover Sheet 
Ywr 11 Age Class 2 



DS 



PSU 



Packages 



""X — 1 
































































































































































































































































































TOTALS 

































Directions: i 

I» Enclose a hardshell cover sheet In each hardshell (box) of the PSU* 

2. Enter each school number Included In the hardshell* 

3. Enter the quantities of the peckagas assessed In each school. • 

4. Total the oolumn for each package contained In the hardshell* 

Disposition: White copy to MRC; yellow copy retained by DS* 
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YEAR 11 Partial 
RECEIVING PROBLEM TALLY Complete 
Age Class Z 

District Supervisor P.S.U. 

E. A. No. 



1. Worksheet 

a. Package counts incorrect.., 

b. Worksheet not returned 

c. Final group size missing • 

d. Miscellaneous 

2. Supplementary Principal's Questionnaire 

a« P.Q. not returned (without explanation) 

3. Background Questions (tailsheet) 

a. Questions 5-9 gridded when specifically not 
allowed by school 

b. Questions 5-9 not answered and no explanation 
included 

c. Question 9 not answered 

d. Question 9 - in U.S. - entry is other than a " 
state 

e. Miscellaneous 

4. Package Condition Ovals (PCO) 

a. Not gridded or double gridded 

b. NR or NE gridded but no explanation on package 
or on school worksheet 

c. P gridded but no explanation- reason and affected 
exercise(s) noted on front of package 

d. N gridded, but administration is partial , 

e. BD is out of range, but PCO not coded as NE 

f. Miscellaneous 35 



5. Group Administration Schedule Forms (Tearoffs) 









1 

1 


b. Returned but incomplete or not legible. . 
































4. Rarp 
















c. Entries transferred incorrectly to front of 
























3. Birthddte 








5. PSU/School Number 






, 








d. ID written incorrectly on GAS 








e. Miscellaneous 









6. Miscellaneous 

a. Incorrect packages administered .' 

b. Hardshell incorrectly packed 

1. Schools are interspersed and not packed as 
cohesive units. 

2. Within the school unit, package numbers are 
mixed together. For example, Package l's 
and 2's are interspersed. 

c. Cover sheet filled out incorrectly or not 
returned 

d. Other miscellaneous 
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6. 



8. 



' Does your family get a newspaper regularly? 
CD Yes CD No CD I don't know. 
Does your family get any magazines regularly? 
CD Yes O No o I don't know. 
Are there more than 25 books in your home? 
<ZZ> Yes CD No ( — ) I don't know. 
Is there an encyclopedia in your home? 
Yes cz> No CZD I don't know. 



IN 7Ht AKtA 
BELOW, 



How much school did your father complete? 

(FILL IN THE ONE OVAL which best shows how much school your 
father completed.) 

Did net complete the 8th grade 

Completed the 8th grade, but did not go to high school 
Went to high school, but did not graduate from high school 
Graduated from high school 
Some education after graduation from high school 



I don't know. 

Did your father graduate from a college or university? 
Yes CD No CD I don't know. 



How much school did your mother complete? 

(FILL IN THE ONE OVAL which best shows how much school your 

mother completed.) 

Did not complete the 8th grade 

Completed the 8th grade, but did not go to high school 
Went to high school, but did not graduate from high school 
Graduated from high school 

Some education after graduation from high school 



I don't know. 

Did your mother graduate from a college or university? 

CD Yes CD No CZD *I don't know. 

Where did you live on your ninth birthday? 

In the United States (Please specify the state or territory.) 



Outside the United States (Please specify the country.) 



<ZZ> I don't know. 



AL CD 
AK 
AZ 
AR 
CA 
CO 
CT 
DE 
FL 
GA 
HI CZD 
ID 
IL 
IN 
IA 
KS 
KY cd 
LA 
ME 
MD 
MA 
MI 
MN 
MS 
MO 
MT 
NB 
NV 
NH 
NJ 
NM 
NY 
NC 
ND 
OH 



OK 
OR 
PA 
RI 
SC 
SD 
TN 
TX 
UT 
VT 
VA 
WAi 
WVc 
WI 
WY 
DC 
OT CD 
BL 



OC 
BL 
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Contents and Order of an Unopened PSU 



GflSU 



^np Km, Scr 



GAS'* 



The number and order may 
vary slightly from DS to DS. 
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ATTACHMENT 9 
OMMITTED DUE TO CONFIDENTIALITY 
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AGE ■ Col. 4 ID = Col. 9-12 

YR AGE PKG ID 



EXERCISES = Cols. 17-50 



PROBLEM 




